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ABSTRACT 

This study describes the multifaceted components of an assessment process. The paper explains a 
novel approach in which an advisory’ council participated in a “fun, ” hands-on activity to rank- 
order learning outcomes. The top ranked learning competency, as identified by the advisory 
council, was the need for students to gain a better understanding of internal controls. With this 
competency identified, the advisory council exercise was then followed-up by a modification in the 
auditing course. An empirical study, consisting of a control group and a treatment group, was 
conducted to assess whether performance on an internal control essay question by students now 
met or exceeded established expectations. The results indicated that students’ preliminary 
understanding of internal controls had been enhanced. The accounting faculty further closed the 
loop by approving a new internal controls course designed to cover, in greater detail, topics such 
as the COSO internal controls framework, Sarbanes-Oxley requirements, recent PCAOB 
statements, and real-world cases involving internal control failures. 
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INTRODUCTION 

ccounting educators are being challenged to engage in meaningful direct assessment evaluations 
(Calderon, Green, & Harkness, 2005; Martell, 2005). A major subject matter in accounting 
education is how to include educators and other stakeholders in the assessment process, and then 
follow up with modifications to the learning process. This study’s contribution to the accounting literature is its 
illustration of an assessment process, from beginning to end. The last stage of an assessment process is often the 
most difficult aspect. As Rotondo (2005, p. 96) explains: 

Closing the loop is the holy grail of assessment activities. The most pristine assessment plan is worthless unless it 
has been used to improve the business curriculum. 

This paper first illustrates a course of action in which an accounting department was able to actively 
involve the professional community, an advisory council, in the role of ranking learning competencies. The top 
learning competency identified by the advisory council was to understand internal controls and how controls can 
ensure the accuracy and integrity of financial data with respect to the safeguarding of business assets. Second, based 
on this top learning competency, a course-embedded assessment tool was developed for an auditing course. This 
tool, which was an essay question on the final exam, was used to evaluate whether students’ answers met or 
exceeded expectations with regard to understanding internal controls. 

Course-embedded assessment is the collection of data, typically by exams or other graded projects within 
the regular classroom environment. As Ammons and Mills (2005) explain, course-embedded assessment offers 
several benefits, the most notable being that students are motivated to respond seriously to the assessment tool since 
it can impact their grades. Therefore, the incorporation of course-embedded assessment via an examination question 
enhances the internal validity of the current study. Not surprising, by expanding the lecture on internal controls in 
the auditing course, the subsequent empirical findings indicated that students had a better understanding of internal 
controls. 
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The assessment study was conducted at an AACSB-accredited business school located in the Southwest 
offering undergraduate and MBA business degrees. The College expects all business majors to develop 
competencies in the areas of written and oral communication, analytical skills, ethics, and self-management. In 
addition to these generic competencies, the accounting department faculty identified ten additional functional 
competencies that an accounting major is expected to have when completing the program. These ten competencies 
are listed in Table 1 in no specific order. 1 


Table 1: Ten Things That an Accounting Major Should Know 


1. Prepare financial statements that are consistent in form & content with current professional standards 

2. Understand GAAP 

3. Apply accounting concepts & principles in practice 

4. Use technology appropriately to research and analyze and/or solve accounting problems 

5. Understand internal controls & how controls can ensure the accuracy & integrity of financial data & safeguarding of 
business assets 

6. Analyze systems & identify the functions for each technology component in a system; design & implement 
appropriate control systems for each component 

7. Understand the role auditing plays in promoting the free flow of reliable information 

8. Comprehend the content, structure, & meaning of reporting for internal operations, & apply the concepts to create 
accounting reports & design management planning & control systems 

9. Understand the role of taxation in business & personal decision-making processes 

10. Understand current accounting theory & the approaches to its development 


While generic skills are important, the advisory council’s focus was on the functional competencies that an 
accounting major is expected to have when completing the program. At the annual spring accounting advisory 
meeting participants engaged in an exercise to identify the importance of the aforementioned ten competencies. 
Each competency was written on flip charts which were randomly distributed around the meeting room. Ten 
professionals, seven faculty members, and one student were each given four “P” dots and 4 “I” dots. The “P” 
represented public accounting and “I” represented industry. The participants then went around the room and placed 
their allotment of dots on what they considered to be the most important competencies. The professionals consisted 
of two audit partners (“Big 4“ and a national CPA firm), two audit managers (one “Big 4“ and the other from the 
state’s Auditor General office), one attorney, one sole practitioner, one chief financial officer, one CPA executive 
from the State’s Society of CPAs, one internal audit director, and one controller. Table 2 shows the results. 

The accounting advisory council members found the activity to be an innovative and inspiring way to 
involve panel members in the assessment process. Elicitation of professionals’ views with respect to rank ordering 
the ten competencies provided the accounting faculty with a unique perspective on their perceived importance to the 
business community. In addition, this procedure touched on some important broad principles related to assessment. 
For example, as Apostolou (1999, p. 179) states: 

■ Assessment fosters wider improvement when representatives from across the educational 
community are involved, and 

■ Assessment makes a difference when it begins with issues of use and illuminates questions that 
people really care about 

From Table 2 it is easy to ascertain that understanding internal controls was the learning competency that 
emerged as most important. With the leading learning competency identified, a final exam essay question was 
developed. As shown in Table 3 this essay question contained three parts. Parts II & III of the essay question were 
adapted and modified from Messier, Glover & Prawitt (2006). 
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Table 2: Evaluation of Functional Competencies 


(Each person had 4 "P" dots & 4 "I" dots) 

AAC Members 
(n=10) 

Faculty (n=7) 

Student (n=l) 



dark blue dots 

yellow dots 

light blue dots 

Total 

Functional Competency 

Public 

Industry 

Public 

Industry 

Public 

Industry 

Dots 

Understand internal controls & how controls 
can ensure the accuracy & integrity of 
financial data & safeguarding of business 
assets 

9 

8 

3 

4 

1 


25 

Understand GAAP 

6 

4 

6 

3 



19 

Prepare financial statements that are 
consistent in form & content with current 
professional standards 

4 

4 

5 

3 

1 

1 

18 

Use technology appropriately to research and 
analyze and/or solve accounting problems 

6 

3 

3 

1 

1 

1 

15 

Comprehend the content, structure, & 
meaning of reporting for internal operations, 

& apply the concepts to create accounting 
reports & design management planning & 
control systems 

1 

8 


5 



14 

Analyze systems & identify the functions for 
each technology component in a system; 
design & implement appropriate control 
systems for each component 


6 

1 

5 


1 

13 

Understand the role auditing plays in 
promoting the free flow of reliable 
information 

6 

1 

4 


1 


12 

Apply accounting concepts & principles in 
practice 

4 

4 

3 

1 



12 

Understand the role of taxation in business & 
personal decision-making processes 


2 

1 

5 


1 

9 

Understand current accounting theory & the 
approaches to its development 



2 

1 



3 


36 

40 

28 

28 

4 

4 



Control Group 

In order to establish a baseline of student understanding of internal controls and determine if an 
improvement in this understanding was attained, an essay question was included on the final exam of the auditing 
course during the fall of 2005. The auditing course is a required course for all accounting majors seeking an 
emphasis in “public accounting.” The essay question, which contains three parts, is shown in Table 3. The first part 
required an objective answer pertaining to the definition and objectives of a system of internal controls. The second 
part required each student to apply the principles of internal controls to a specific transaction processing system, the 
expenditure cycle. The third part of the question required analysis, in that students’ were requested to identify the 
consequences of internal control weaknesses in the expenditure cycle. 

Treatment Condition 

In the spring 2006 auditing class the curriculum was modified to include an extended lecture on internal 
controls in the expenditure cycle. The objective of providing additional coverage was to enhance students’ 
understanding of internal controls and to close the loop, so to speak, on any deficiencies in their learning outcomes. 
The extended lecture included more discussion of understanding the dimensions and the definition of internal 
controls, as well as a greater emphasis on appreciating what key duties and jobs need to be segregated. 
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Table 3: Essay Question - Internal Controls 


Questions'. 

Part I: Define internal control and the objectives of a system of internal controls 

Part II: List the key segregation of duties in the expenditures cycle - the request, ordering, and payment for merchandise 
inventory 

Part III: What errors or fraud can occur if the duties in the expenditure cycle are not segregated? 

Suggested Answers: 

Part I: Internal control is the set of policies and procedures developed to ensure the safeguarding of an entity’s assets, the 
reliability of its accounting records, and the accomplishment of overall company objectives. The objectives of a system of 
internal controls are to provide reasonable assurance regarding 

• Reliability of financial reporting 

• Effectiveness and efficiency of operations 

• Compliance with applicable laws and regulations 

Part II: 

• The purchasing function should be segregated from the requisitioning and receiving functions 

• The invoice-processing function should be segregated from the accounts payable function 

• The disbursement function should be segregated from the accounts payable function 

• The accounts payable function should be segregated from the general ledger function 
Part III: 

• Theft of goods 

• Possible payment for unauthorized purchases 

• Theft of cash 

• Overpayment for goods and services 

• A defalcation that would normally be detected by reconciling subsidiary records with the general ledger control 
account 


Scoring the Examination Questions - Independent Variable 

A total of 21 students completed the fall 2005 auditing course, took the final examination and provided an 
answer to the internal control essay question. In the spring, 2006 auditing course, a total of 26 students completed 
and provided an answer to the internal control question on the final examination. To enhance internal validity in the 
scoring of the internal control questions, the 47 student responses were photocopied and randomly sorted using a 
random number table. In addition, the student names did not appear on any of the photocopied responses. 
Therefore, when scoring the essay questions, the evaluators did not know if the question responses were from the 
control or from the treatment groups. 2 This provided a reasonable control against bias in the scoring of the student 
responses and supports a reasonably clean assessment of the effect of the treatment condition on student learning. 

Each part of the three-part essay question was subsequently evaluated independently by two accounting 
educators using the rubric as described in Table 4. Using agreed-upon rubrics is recognized by the AACSB as an 
effective approach to evaluate learning competencies for assessment purposes (Calderon, Green, & Harkness, 2005). 
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Evaluator #1 was an accounting educator who did not teach the auditing course, but nevertheless had an in-depth 
understanding of internal controls. Evaluator #2 was the instructor of the auditing course. 


Table 4: Assessment Rubric 



Below Expectations 

1 

Meets Expectations 

2 

Exceeds Expectations 

3 

Define internal control and the 
objectives of a system of internal 
controls 

Identifies fraud 
prevention as a goal and 
lists no objective. 

Identifies management’s 
role and lists one 
objective. 

Identifies management’s 
role and lists two or more 
objectives. 





List the key segregation of duties in 
the expenditures cycle - the 
request, ordering, and payment for 
merchandise inventory 

Identifies the segregation 
of no or one pair of 
functional areas. 

Identifies the segregation 
of two pairs of functional 
areas. 

Identifies the segregation 
of more than two pairs of 
functional areas. 





What errors or fraud can occur if 
the duties in the expenditure cycle 
are not segregated 

Identifies no or one error 
or fraud. 

Identifies two errors or 
fraud. 

Identifies more than two 
possible errors or fraud 
and links the errors or 
fraud to the functional 
pair that should be 
segregated. 


Each evaluator assigned a score on a three part scale. A score of “1” was assigned if the student did not 
“meet expectations” of that expected of someone with an understanding of internal controls (i.e., the student did not 
answer the question correctly as specified by the criteria in the rubric). A score of “2” was assigned if the student 
“met expectations” of someone with an understanding of internal controls (i.e., the student answered the question 
mostly correct as specified by the criteria in the rubric). A score of “3” was assigned if the student “exceeded 
expectations” of someone with an understanding of internal controls (i.e., the student provided an excellent answer - 
completely correct). 

After independently scoring the 47 student responses, the two evaluators convened to discuss any 
differences in the resulting scores. Where differences occurred between the evaluators, the individual student’s 
response was examined and rescored so that a consensus score was applied to the response. The frequencies of 
agreement between the two evaluators were 77%, 59%, and 66% for parts (1), (2), and (3), respectively. More 
importantly, the statistical analysis performed on the data set generated from each evaluator was consistent with the 
consensus results as reported in Table 5. 

RESULTS 

Table 5 displays the results of student performance on the internal control question across both semesters; 
fall semester (control group) and spring semester (treatment group). The data displays the number and 
corresponding percentage of students’ scores at each of the three performance levels (below expectations, meets 
expectations, and exceeds expectations) on each of the three parts of the essay question. 

When viewing the data in the top part of Table 5 we found the results of the fall semester (control group) 
somewhat discouraging. Only 29% of the students met expectations on the first part of the essay question and no 
students exceeded expectations. 3 

Performances on the other two parts of the question were better, in that about 75% of the students met or 
exceeded expectations. This divergence of results is interesting in that most students could not well articulate the 
definition or objectives of a system of internal controls (the answer required for part (1) of the essay question), but 
they could describe which controls should be in place within the expenditure cycle. 
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Table 5: Results of Assessment Evaluations - Number (%) of Students 


Scored at Each Performance Level 


Question Part 



Part 1 

Part 2 

Part 3 

Fall Semester - Control (n = 21) 

Below Expectations 

15 (71%) 

5 (24%) 

6 (28%) 

Meets Expectations 

6 (29%) 

14 (67%) 

14(67%) 

Exceeds Expectations 

0 ( 0%) 

2 ( 9%) 

1 ( 5%) 

Spring Semester - Treatment (n = 26) 

Below Expectations 

1 ( 4%) 

5 (19%) 

6 (23%) 

Meets Expectations 

9 (35%) 

11 (42%) 

18 (69%) 

Exceeds Expectations 

16 (61%) 

10 (39%) 

2 ( 8%) 


Key: 

1 - Did not meet expectations, i.e., did not answer the question(s) correctly 

2 - Met expectations, i.e., pretty much answered the question(s) correctly 

3 - Exceeded expectations, i.e., gave an excellent answer - completely correct 


Comparing the percentages between the fall semester students (control group) and the spring semester 
students (treatment group) indicates a performance improvement, at least on the first two parts of the essay question. 
For part (1), performance increases showed that 96% of the students met or exceed expectations. For part (2), 
performance increases showed that 81% of the students met or exceed expectations. Most of the increase in 
performance for part (2) resulted from a higher proportion of students who exceeded expectations (from 9% to 
39%). The data in Table 5 indicates little difference in performance across groups for part (3) of the essay question. 

A binomial z-test was carried out on the data and revealed statistically significant performance changes (p 
< .05) across groups for the student responses to part (1) and part (2) of the essay question. 4 Specifically, the 
percentage of students who scored below expectations for part (1) declined from 71% to 4%, while the percentage of 
students who exceeded expectations on part (1) increased from 0% to 61%. For part (2) of the essay question the 
percentage of students who exceeded expectations increased from 9% to 39%. No statistical difference was 
discovered for student performance on part (3) of the essay question. 

LIMITATIONS AND CONCULSIONS 

The assessment process that we describe is indeed multifaceted. Input was obtained from professional 
accountants serving on an advisory board and, based on this input the curriculum pertaining to internal controls was 
modified in an auditing course. In addition, performance by students was assessed. Our empirical findings only 
provide preliminary evidence that may be used to close the loop on assessment. While the interventions we 
implemented were time consuming and costly, especially when considering the opportunity costs of the professional 
accountants serving on the accounting area's advisory council, they must be viewed in the context of continuous 
improvement necessary for a more robust assessment. 

Modifying the auditing course was the first stage of the assessment process. The final closing the loop took 
place when a new internal auditing & controls course was offered in the spring 2008 semester. This new course 
covered internal controls topics in depth. The course used an internal auditing textbook, published by the Institute 
of Internal Auditors (Reding, et. ah, 2007), a case book by Michael Knapp (2006), and a paperback titled “Sarbanes- 
Oxley for Dummies” by Jill Welytok (2006). As Calderon, Green, & Harkness (2005, p. 282) explains: 
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The final part of the process - using the evidence to initiate learning improvement opportunities - closes the loop. 
This process of closing the loop means that the program has completed all fundamental activities in the assessment 
process and is now ready to employ the evidence to initiate course and curriculum changes. Closing the loop 
addresses the most fundamental goal of assessment - it assures that assessment leads to program improvement. 

The findings from the essay test showed that there was some improvement in students’ understanding of 
internal controls. However, a limitation of the empirical test is that it only “scratches the surface” of understanding 
internal controls. The key evidence to initiate learning improvement, especially the driving force to develop a new 
course, was the overall input that was obtained from professional accountants serving on an advisory board. 

Feedback from experts on assessment suggests that it can be a real challenge to overcome faculty resistance 
to assessment (Martell, 2005). In this setting, our primary contribution lies in documenting the multifaceted 
components fundamental to the assessment process to enhance curriculum improvements. 

NOTES 

1 Ammons and Mills (2005, p. 5) argue that instead of objectives that use verbs like “understand” or “appreciate,” 
objectives will provide a better basis for assessment if they use observable descriptions of learning outcomes with 
action verbs like “calculate,” “categorize,” “compare,” or “construct.” 

2 The same essay question included on the fall, 2005 final examination, and shown in Table 3, was included on the 
final examination for the students enrolled in the spring, 2006 course. The final examinations administered during 
the fall, 2005 semester were not returned to students in order to prevent knowledge transfer of the question to 
students enrolled dining the spring, 2006 semester. Students in both semesters automatically received full credit 
for this essay question to ensure fairness for all students. Not grading this question allowed clean and unmarked 
photocopies of students’ answers to be available for the evaluators. Using the identical question in the spring 
allowed a comparison of student performance as a result of the treatment condition. 

3 A question often asked by educators is what percentage of students must meet, or exceed, expectations in order to 
be satisfied that standards are being met. The Association to Advance Collegiate Schools of Business (AACSB) 
provides this guideline: “One emphasis in the Assurance of Learning Standards is to gather data on student 
learning to be used for the purposes of improving business curricula. For each learning goal, the school’s faculty 
will determine its minimum expectation or standard. There is no prescribed percentage of students that must meet 
the standards articulated by the faculty. What the review team will be looking for, instead, is how these data are 
used. Thus, a poor showing on student mastery of a learning goal (e.g., analytical reasoning) would only be a 
concern if the curriculum was not subsequently modified to improve student skills in this area. A second purpose 
for the learning goals is to communicate the competencies of graduates to students and employers. Thus, the 
goals should represent learning goals achieved by nearly all graduates, not just a portion. The review team will 
also examine whether the school’s performance standards are appropriate given the student body demographics 
and the school’s mission.” (see: www.aacsb.edu/resource_centers/assessment/frequently-asked.asp). 

4 Could the results be driven by the possibility that students in the spring course were better students then those 
enrolled in the previous fall semester course? To eliminate this alternative explanation, we examined final exam 
scores without the essay question. Final exams were not distributed back to students and the same exam was used 
in both the fall (control) and spring (treatment) semester. The mean, variance, and median for the control versus 
treatment groups were 66.2, 148.0, 70, and 63.0, 123.6, 65, respectively. Results from a t-test on the final exam 
scores indicated that there was no significant difference between the two classes (p = .38). 
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